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ABSTRACT:  This  document  is  a  combined  test  plan  and 
test  report  for  the  CAMPS  Test  1-32 .  The 
test  investigated  the  benefits  of  the  CAMPS 
to  the  mission  planning  process  using  prob¬ 
ability  of  survival  as  a  measure  of  perfor¬ 
mance.  In  addition,  data  was  collected  and 
analyzed  to  assess  the  utility  of  the  CAMPS 
as  currently  configured.  Test  design  and 
administration,  data  management  and  analysis, 
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EXECUTIVE  SUMMARY 


This  document  is  a  combined  test  plan  and  test  report  for  the  CAMPS  test 
1-62.  The  purpose  of  the  CAMPS  Test  1-32  was  to  investigate  the  mission 
planning  benefits  and  the  utility  of  CAMPS,  a  Computer-aided  Mission  Planning 
System. 

There  were  two  facets  to  the  CAMPS  Test  1-82.  One  was  the  investigation 
of  the  benefit  arising  from  the  use  of  the  CAMPS  over  the  current  manual 
technique.  The  other  was  the  collection  of  data  to  estimate  the  utility  of 
the  CAMPS  as  it  is  currently  configured. 

In  order  to  investigate  the  benefits  of  the  CAMPS  to  the  mission  planning 
process,  tactical  flight  routes  were  developed  by  test  participants  using  both 
manual  techniques  and  using  the  CAMPS.  Each  of  these  flight  routes  was  then 
evaluated  using  a  system  called  the  Experimental  Penetration  and  Analysis 
Support  System  (EPASS).  A  probability  of  survival  for  each  mission  was 
derived  using  the  probability  of  sustaining  abort  level  damage  provided  by  the 
EPASS.  The  difference  in  the  probabilities  of  survival  for  missions  planned 
with  the  CAMPS  and  missions  planned  manually  was  then  computed  for  each 
participant  and  used  as  the  measure  of  benefit  provided  by  the  CAMPS. 

The  utility  of  the  CAMPS  was  investigated  using  the  subjective  responses 
of  the  test  participants  on  various  questionnaires.  The  results  of  the 
questionnaires  were  summarized  and  presented  in  the  appendices. 

The  key  result  of  the  CAMPS  Test  was  that  analysis  of  the  resulting 
probabilities  of  survival  made  it  possible  to  reject  the  hypothesis  that  CAMPS 
planning  is  not  more  effective  than  manual  planning  at  the  following  levels  of 
significance: 

To  the  Target  p  -  .052 

Total  Mission  p  *  .052 

What  this  and  the  data  imply  is  that  an  improvement  in  the  probability  of 
survival  of  about  25?  can  be  anticipated  when  using  the  CAMPS  as  compared  to 
manual  methods. 

Overall  the  system  was  well  received  by  the  naval  aviators  and  naval 
flight  officers  of  MAG-15  at  El  Toro,  California.  The  utility  of  the  system 
was  rated  very  high  with  suggestions  for  improvement  being  offered  and 
detailed  in  the  report. 
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‘  .  *  3ACXGECUC7I 
'  . '  .  ‘  General 

The  Marine  Corps  is  currently  involved  in  the  procurement  of  a  prototype 
automated  tactical  aircraft  mission  planning  aid  called  CAMPS  'Computer  Aided 
Mission  Planning  System).  It  is  the  intention  of  the  Marine  Corps  to  use  thi 
system  for  training  and  exercises  in  order  to  more  clearly  define  the 
reauirements  for  such  a  system  in  the  Fleet  Marine  Force  FMP' .  It  is  to  thi 
end  that  the  test  i escribed  in  this  document  was  designed  and  conducted.  It 


is  one  in  a  series  of  evaluation  ef  f  " — « 


s  envisionec 
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of  CAMPS  needed,  and  redefining  these  areas  requiring  modification  before 
imnlementation  in  the  Marine  Corns. 


’.’.2  lescrinticn  of  CAMPS 


CAMPS  is  a  microcomputer-based  system.  It  uses  color  graphics  technclo 
a  terrain  elevation  data  base,  and  enemy  surface-to-air  missile  ,3AM'  data 
’currently  ’unclassified)  to  present  to  the  tactical  mission  planner  a  view 
the  enemy  SAM  threat.  This  depiction  of  the  SAM  threat  has  been  modified  t 
indicate  the  effects  of  terrain  masking  and  aircraft  altitude  on  the 
capabilities  of  the  SAMs.  To  develop  a  flight  plan,  the  planner  designates 
altitude  and  is  then  presented  a  view  of  the  enemy  SAM  threat  at  that 
altitude.  He  then  plans  a  route  'ey  specifying  checkpoints  that  allow  him  t 
avoid  the  greatest  concentration  of  enemy  threat  as  indicated  by  the  thr?.  t 
boundaries  (indicating  effect  of  terrain  masking)  and  the  color  coding 
areas  around  the  sites. 


3 


Upon  completion  of  a  route,  the  TAMPS  computes  the  coordinates  of  the 
checkpoints,  time  checks,  and  fuel  usage  data,  then  prints  this  route 
information  in  a  kneefcoard  card  format. 


Threat  capability  parameters,  aircraft  performance  specifications, 
terrain  data  base  can  be  varied  to  support  operations  in  any  scenario. 


arc 


1.1.3  Previous  Tests 


Previous  tests  using  the  CAMPS/EPASS  or  earlier  versions  of  the  same 
system  consisted  of  a  manual/automated  test  conducted  under  tasking  from  the 
Marine  Corps  Operations  Analysis  Group  (MCCAG).  The  purpose  of  the  test  was 
to  measure  the  benefit  of  automation  in  mission  planning,  both  for  aviation 
and  ground.  The  results  were  used  for  a  Cost  and  Operational  Effectiveness 
Analysis  ( COEA )  of  the  Tactical  Combat  Operations  (TCO)  System.  CAMPS  itself 
was  not  tested,  however  it  was  used  as  a  device  to  simulate  an  automated  aid 
to  planning.  Its  objectives  were  very  similar  to  Objective  1  of  this  test 
although  the  testing  procedure  was  radically  different. 


A  first  effort  at  evaluating  CAMPS  itself  was  recently  conducted  at  Marin 
Aircraft  Wing  Training  Squadron-l  (MAWTS-1 )  at  Yuma ,  Arizona;  where  data  was 
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on  the  mission  planning  benefits  and  the  utility  of  the  OAP!PS  [23  currently 
configured)  against  a  specified  threat  level.  Zaza  will  be  used  as  a  basis 
for  planning  of  future  tests  and  further  development  of  OA.'dPS.  lata  is  meant 
to  be  combined  with  the  test  results  of  future  testing  in  order  to  contribute 
to  decisions  concerning  procurement . 


TEST  OBJECTIVES 


Objective  ■ 


.3  investigate  whether  a  plan  for  a  tactical  air  3 trike  mission  developed 
using  OA.'IPS  results  in  a  higher  probability  of  survival  than  one  developed 
manual I7 . 


'.3.2  Objective  2 


To  assess  test  participant  attitudes  regarding  the  number  of  required 
altitude  slices  and  the  specific  al tirades  preferred. 

*  .3-3  Objective  3 

To  assess  test  participant  attitudes  regarding  the  physical  appearance  an 
information  content  of  the  OAOJPS  display. 

1.3.4.  Objective  i 

To  assess  the  user  compatibility  of  the  current  0A0!?S  lease  board  card. 

‘  .J .5  Objective  5 

To  assess  the  ease  of  planning  -with  OA.MPS. 

'.J.6  Objective  5 

To  assess  the  marginal  value  of  adding  Experimental  Penetration  and 
Analysis  Support  System  (2PASS)  type  capabilities  to  the  0  A  PIPS . 

'.3.7  Objective  ~ 

To  assess  the  user's  overall  opinion  of  the  0 A.MPS/EPASS  system  as  a 
mission  planning  tool . 

1  .3.3  Objective  3 

To  compile  a  list  of  participant  likes  and  dislikes  about  the  OA.MPS/EPASS 
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2.'  APPROACH 

There  were  two  facers  ro  the  CAMPS  Test  '-52.  One  was  rhe  investigation 
cf  rhe  henefir  arising  iron  rhe  use  cf  rhe  TAMPS  over  rhe  current  manual 
technique.  The  other  was  rhe  collection  of  data  ro  esrinare  rhe  utility  of 
the  CAMPS  as  it  is  currently  configured. 

In  order  ro  investigate  rhe  benefits  of  rhe  TAMPS  ro  rhe  mission  planning 
process,  tactical  flight  routes  were  developed  'ey  rest  participants  using  both 
manual  technioues  and  using  rhe  CA.MPS.  Pach  of  these  flight  routes  was  then 
evaluated  using  another  TCMARCC,  Inc.  system  called  the  Experimental 
Penetration  and  Analysis  Purport  System  (2PASS).  Tetails  of  the  evaluation 
procedure  are  contained  in  Appendix  A.  A  probability  of  survival  for  each 
mission  was  derived  using  the  rrobability  of  sustaining  abort  level  damage 
provided  'ey  the  2? ASS.  The  difference  in  the  probabilities  of  survival  for 
missions  planned  with  the  TAMPS  and  missions  planned  manually  was  then 
computed  for  each  participant  and  used  as  the  measure  of  benefit  provided  by 
the’ TAMPS. 

The  utility  of  the  TAMPS  was  investigated  using  the  subjective  responses 
of  the  test  participants  on  various  questionnaires.  The  results  of  the 
questionnaires  were  summarised  and  used  to  meet  Objectives  2  through  5. 

2 . 2  FACTORS 

The  factors  and  the  levels  of  each  factor  in  parentheses,  considered  in 
the  design  of  the  CAMPS  test  were: 

Planning  System  'TAMPS : Manual) 

■Order  of  Testing  (Manual, '.CAMPS)  (CAMPS/Manual ) 

Test  Participant  ('-24-' 

Target  ( 1 ,2) 

Level  of  Threat  (Constant) 

Aircraft  TJsed  (Constant) 

Intelligence  P.eliability  (Constant) 

2.2.1  Planning  System 


As  Objective  1  states,  the  !<ey  point  of  investigation  is  the  difference 
between  the  TAMPS,  an  automated  system,  and  the  manual  system  of  mission 
planning.  Therefore,  the  type  of  planning  system  used  to  prepare  a  flight 
route  was  chosen  as  the  dependent  variable  for  Objective  1 . 

2.2.2  Order  of  Testing 

The  two  levels  of  this  factor,  the  CAMPS/Manual  order  and  the  Manual, 'CAMPS 
order,  were  distributed  in  the  test  layout  in  a  balanced  manner.  This  was 
accomplished  by  having  the  odd-number  participants  use  the  CAMPS/Manual  order 
and  the  even-number  participants  use  the  Manual/CAMPS  order.  Both  orders  were 
used  in  an  effort  to  reduce  the  effect  of  learning  associated  with  the 
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The  greatest  degree  of  variability  vas  anticipated  among  the  test 
participants.  This  vas  controlled  "07  having  each  participant  act  as  his  own 
control  and  by  the  number  of  test  participants  used.  Table  2-‘  3hows  the 
distribution  of  the  2d  test  participants  amongst  the  various  combinations  of 
factors. 


Table  2-1 .  Test  lay-out 


Planning  System  — > 

TAMPS 

XANUAL 

CAMPS / 

MANUAL/ 

MANUAL/ 

CAMPS/ 

Order  of  Testing  — > 

MANUAL 

CAMPS 

CAMPS 

MANUAL 

1 

4 

4 

, 

5 

3 

3 

X 

Target 

9 

12 

12 

9 

1 

«  T 

•  J 

'  6 

4  6 

-  ? 

1" 

2C 

2C 

1 

21 

24 

24 

✓  4 

j 

2 

2 

T 

'■* 

5 

6 

*■» 

Target 

4  4 

10 

*c 

•  * 

2 

*  > 

14 

•4 

«  e; 

•  9 

18 

4  s 

•  Q 

2J 

22 

22 

23 

2.2.4.  Target 


Since  each  test  participant  planned  tvo  flight  routes,  it  vas  necessary  to 
have  two  different  targets  with  different  starting  points  so  that  each 
planning  effort  might  produce  a  different  route.  Tetails  of  the  targets  used 
can  be  found  in  Appendir  3.  The  two  targets  were  assigned  in  a  balanced 
manner  so  that  a  planning  system,  order  of  testing,  3nd  target  combination 
occurred  with  equal  frequency. 
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The  difference  in  the  probability  of  survival  between  3  A. TPS  and  nanual 
plana)  from  the  start  of  the  mission  to  the  target. 

2.3  ■■  .2  M3?  1  b 


tr.e  star- 


The  difference  in  the  probability  of  survival 
mission  to  the  end  of  the  mission. 

2.3.2  Objective  2 

2. 3. 2.1  >SP  2a 


Test  participant  opinions  of  the  minimum  number  of  altitude  sli 
required  to  perform  mission  planning. 

2. 3. 2. 2  M3?  2b 

The  "Relative  Worth"  of  altitude  slices  as  determined  from  test 
participant  preferences  under  diminishing  resources . 

2.3*3  Objective  3 


093 


,2. 3. 3-1  M3?  3a 

Test  Participant  categorical  judgements  of: 

—  color 

—  information  content 

—  line  texture 
size 

—  scales 
overall  display 

2. 3. 3.2  MOP  3b 

Test  participant  subjective  comments  on  the  CAMPS  display. 


Test  participant  ranking  of  the  current  kreebcard  card  and  crocosed 
alternative  kneecoard  cards . 

2 .3.1.2  >D?  £b 

Test  participant  subjective  comments  on  the  format  and  content  of  the 
kneeboard  cards . 

2.3.5  Objective  5  '  MO?  5' 

Test  carticinant3  categorical  judgements  of  overall  ease  of  0 banning  vie 
the  0 AMPS'. 

2.3.6  Objective  6  '.’DP  6' 

2.3.T  Objective  ~  (MO?  7) 

Test  participant  categorical  judgements  of  the  CAMPS/EPA SS  system. 

2  . 3  -3  Objective  3  (MOP  3) 

Test  participant  comments  on  the  aspects  of  the  systems  that  they  liked 
and  disliked. 

...4.  SCENARIO 

The  sensitivity  of  the  experiment  to  the  enemy  threat  vas  identified 
early.  Too  veak  of  a  threat  would  allow  anyone  using  manual  or  TAMPS 
procedures  to  maneuver  'unscathed.  Too  heavy  of  a  threat  would  allow  no  one 
survive  no  matter  how  well  the  mission  was  planned.  So,  selection  of  a  thre 
became  a  critical  issue.  It  was  decided  that  the  scenario  and  threat 
developed  for  the  final  exercise  at  the  Weapons  Training  Instructor  ’  VTI ' 
Course  held  at  Marine  Aircraft  Wing  Training  Squadron-1  (MAWTS-t),  Yuma, 
Arizona,  would  be  used.  A  detailed  summary  of  the  scenario  and  threat  is 
contained  in  Appendix  3. 

2.5  TEST  PARTICIPANTS 

The  24.  test  participants  for  the  OAMFS  Test  1-82  were  drawn  from  Marine 
Aircraft  Croup-1  3  at  Marine  Corps  Air  Station,  El  Toro,  California. 
Requirements  for  participation  were  that  the  individual  be  a  fixed-wing 
aviator  or  a  naval  flight  officer,  currently  assigned  to  flight  status,  with 
experience  in  planning  tactical  missions  desired.  There  were  no  rank 
limitations  imposed  statistics  for  the  population  of  test  participants  used 
for  the  test  which  are  given  in  Table  2-2*. 
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ORGANIZATION 


The  administration  of  the  TAMPS  Test  '-32  was  organised 
Testing,  Evaluation,  Tata  Reduction,  and  Analysis.  Figure 
phases,  their  functions,  and  the  inputs/' outputs  of  each. 


pnases 

these 


3 .2  SCHEDULE 


3-2.1  Testing  Phase 

CAMPS  Test  1-32  was  conducted  during  the  week  of  ’  2-'  5  April  '932.  Each 
day  ox'  testing  was  divided  into  three  periods,  scheduled  as  follows: 

Period 


C"3C-1 1 3C 
1030-14-30 
1330-1730 


Each  period  consisted  of  four  hours  of  activity  broken  down  in  the 
following  manner: 


Activi ty 

Minute 

Introductory  Remarks 

0-2C 

Mission  3rief 

21-30 

Planning  Period  1 

31-90 

Planning  Period  2 

91  -1  50 

Demonstration 

151 -180 

Debrief 

181 -240 

An  outline  of  each  activity  is  contained  in  Appendix  C . 

There  was  a  one-hour  overlap  in  scheduling  between  Periods  I  and  II,  and 
Periods  II  and  III.  This,  overlap  allowed  a  more  effective  use  of  the  TAMPS 
equipment  and  thus  more  observations  in  a  shorter  time  frame. 

Testing  began  with  Period  II  on  Monday  and  concluded  with  Period  I  on 
Friday.  Table  3-1  contains  a  detailed  schedule  of  test  participant  assignme 
to  the  test  periods . 


m 

X 

able 

3-1  .  Par 

ticipant  Schedule 

MON 

TUES 

'VED  THUF.S 

FRI 

II  III  I 

TT 

III  I 

II  III*  I  II 

1  U»*k. 

- 

1  3  5 

n 

9  11 

13  15  17  19 

21 

23 

2  4  6 

3 

10  12 

14  16  IS  20 

22 

24 

*  Due  to  VI?  demonstrati 
planning,  however,  they 

on, 

did 

participa.n 
respond  to 

ts  during  thi3  period 
questionnaires . 

did 

not 
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Fip,iirf«  3-1.  CAMPS  Tnst  1-82  OrRnnization 


3.2.2  Evaluation  Phase 


The  flight  routes  resulting  from  testing  were  evaluated  on  30  April  and  3 
May  1982  at  CO  MARCO,  Inc.,  Anaheim,  California. 

3.2.3  Reduction  and  Analysis  Phases 

Data  from  the  CAMPS  Test  was  reduced  and  analyzed  during  the  period  10-18 
May  1982. 

3.3  PERSONNEL  REQUIREMENTS 

Personnel  required  for  the  CAMPS  Test  were  all  ferine  Officers  provided 
from  the  Command  Systems  and  Analysis  Sections  of  MCTSSA,  except  where  noted 
below. 

Personnel  requirements  were  as  follows: 

a.  Test  Design  and  Preparation  3 

b.  Test  Supervision  and  Data  fenagement  2 

c.  Test  Participants  (MAG-13  El  Toro)  24 

d.  Data  Evaluation  and  Analysis  2 

3 .4  MATERIAL  REQUIREMENTS 
3.4.1  Testing  Phase 
3.4.1  .1  Facilities 


The  following  spaces  were  provided  by  MAG-13  for  the  test: 

Testing  Office 
Briefing  Room 
CAMPS  Planning  Room 
fenual  Planning  Room 

3. 4. 1.2  Hardware 

The  hardware  used  for  the  test  was  the  Convergent  Technology  suite  of 
equipment  being  used  for  the  CAMPS.  Figure  3-2  shows  the  hardware  used  for 
the  test. 

3-4.1  .3  Software 

The  software  for  the  test  was  Version  4.2  of  CAMPS  as  provided  by  COMARCO, 

Inc . 

3. 4. 1.4  Data  Base 


There  were  three  data  bases  used  for  the  CAMPS  Test  1-82. 
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Each  test  participant  was  giver,  a  brief  demonstration  of  the  basis 
capabilities  of  the  SAPPS  during  the  introductory  remarks  .  They  then  received 
a  detailed  explanation  and  hands-on  session  for  those  functions  specifically 
required  for  the  C AITS  planning  period,  lastly,  a  detailed  brief  an 
demonstration  of  ail  the  SAPPS  capabilities  as  well  as  selected  SPAS 
functions  was  presented  to  each  pair  of  test  participants . 

3-5.2  Evaluation  Phase 

In  order  to  evaluate  the  flight  routes  developed  during  testing,  owe 
JCTSSA  officers  received  approximately  eight  hours  of  training  and  hands-on 
experience  in  using  the  SPASS  at  IOL4P.CC,  Inc. 


tn  u- 


'  '■n  ^ 


4.1  DATA  SOURCES 

Data  was  collected  for  evaluation  and  analysis  in  support  of  the  test 
objectives.  Data  was  collected  from  the  test  participants  by  a  combination  of 
flight  route  summary  sheets,  written  responses  to  questionnaires ,  and  oral 
comments • 

1.2  OBJECTIVE  1 

4.2.1  MOP  la 


4.2.1  . 1  Data  Collection 


The  raw  data  for  this  MOP  was  collected  using  the  knee board  card  produced 
by  the  CAMPS  and  by  a  Manual  Plight  Route  form.  Copies  of  both  are  contained 
in  Appendix  E . 

4. 2. 1.2  Data  Evaluation 


All  flight  routes  developed  during  the  testing  phase  were  then  entered 
into  the  EPASS  terminal  and  evaluated  against  the  threat  devised  for  the 
scenario.  For  each  route,  the  EPASS  simulated  engagements  by  enemy  air 
defenses  along  the  route.  For  each  engagement,  the  enemy  site  firing,  the 
time  of  the  engagement  and  the  probability  of  sustaining  abort  level  damage, 
?(d),  were  then  recorded  on  a  Route  Evaluation  Data  Collection  and  Reduction 
Form  (Appendix  S)  .  Dr.  addition,  the  time-on- target  (TOT)  was  noted  and 
recorded . 


4. 2. 1.3  Data  Reduction 


For  each  engagement  along  a  route  planned  by  a  participant  ( there  may  have 
been  none),  the  probability  of  survival,  ?(s),  for  that  engagement,  was 
computed  as 


? ( s )  -  1-?(d) 


using  the  P(d)  for  the  engagement.  The  resulting  ?(s)  for  each  engagement  was 
then  recorded  in  column  5  of  the  Route  Evaluation  Data  Collection  Form. 
Finally,  the  probability  of  surviving  to  the  target,  P*1.3)*  was  computed  oy 
multiplying  together  the  ?(s)  for  all  engagements  with "an  engagement  time 
eaual  to  or  earlier  than  the  TOT.  This  ?*(s)  was  then  recorded  at  the  bottom 
of  the  collection  form.  At  the  end  of  th$  evaluation  phase,  the  ?fc(g)'3  of 
all  participants  were  consolidated  on  the  Objective  1  (MOP  'a)  Data  Summary 
Sheet  and  used  as  data  for  the  analysis  phase  of  Objective  1 . 

4.2.2  MOP  1b 


The  data  management  for  MOP  lb  was  the  same 
for  all  engagements  were  multiplied  together  to 


as  for  MOP 
get  a  ?(s) 


la  except  the  ?(s) 
for  the  mission. 
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4  .J 

4.3.1  :-p?  2a 

4. 2- 1.1  Data  Oollection 

The  data  for  this  >EP  consisted  of  the  nun be r  of  slices  circled  on  the 
Altitude  Slice  Questionnaire  (Appendix  3)  as  being  the  minimum  number  of 
slices  required  to  do  mission  planning  effectively. 

4.2.1  .2  Data  Reduction 

?or  each  test  participant,  a  tail y  consisting  of  his 
number  ( a .g . , (^)  )  was  entered  on  the  Data  Summary  Sheet 
number  of  altitude  slices  indicated .  Finally,  the  total 
each  number  of  slices  was  recorded. 

4.2.2  tt)P  2b 

4. 2-  2. i  Data  Collection 

Data  collection  for  ®P  2b  consisted  of  the  listing,  by  the  test 
participant,  of  those  altitudes  he  would  choose  if  a  specified  number  of 
altitude  slices  were  available  to  him.  These  altitudes  were  listed  for  an 
availability  of  2  through  10  altitude  slices  on  the  Altitude  Slice 
Questionnaire  (Appendix  3) . 

4. 2-  2. 2  Relative  Worth 

The  relative  worth  measure  (w) ,  as  defined  for  this  test,  is  based  on  the 
premise  that  selection  of  an  item,  when  the  number  of  possible  selections  is 
small,  implies  that  a  higher  value  is  placed  on  that  item  than  selection  in 
the  case  where  a  greater  number  of  selections  is  possible.  Por  example,  if 
you  only  get  three  wishes,  then  each  wish  chosen  would  presumably  have  a 
greater  value  to  you  than  if  you  had  ten  or  twenty  wishes  available  . 

To  compute  the  relative  worth  measure,  we  have  a  number  of  different 
possible  levels  of  selections  available  and  a  list  of  items  to  fill  the 
selections  at  each  level .  This  allows  us  to  then  assign  a  series  of  weights 
to  each  item  based  on  the  different  selection  levels  it  was  assigned  to. 

These  weights  consist  of  the  inverse  of  the  selection  level  for  each 
assignment  of  an  item.  If  an  item  is  listed  as  one  of  three  possible 
selections  at  level  three,  each  item  would  get  a  weight  of  one- third  assigned 
for  that  particular  level. 

As  an  example,  let's  assume  there  are  four  ievels  of  selection  available: 
1,  2,  3.  and  4.  Each  level  is  addressed  separately  and  items  A,  3,  C,  and  D 
are  listed  to  fill  the  available  selection  slots  at  each  level.  One  possible 
allocation  looks  like  this: 


circled  participant 
'Appendix  3)  for  the 
number  cf  tallies  fc 
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A 


The  relative  worth,  w,  for  each 

itan  would 

be  ccapu 

ted  as  follows : 

w,  *  .30  - 

'jj  -.25  *  " 

,:a 

wl  -  -23  = 

•  25 

w,  *  1  .0  - 

.50  *  .55  - 

3  i • 

18 

w^  *  .33  - 

,25  *  .53 

As  can  be  seen,  the  weight  for 

an  It  err.  incr 

eases  as 

its  selection  n:e 

iia 

inished  resources  increases . 

s . 7 

.2.3  Data  Reduction 

Using  the  Objective  2  (OTP  2b) 

lata  Reduced 

cn  ttt 

contained  in  Attend 

2,  a  relative  worth  was  computed  for  each  altitude  listed  by  a  test 
participant.  These  values  were  then  consolidated  and  grouped  around  index 
altitudes  as  follows: 

listed  Altitudes  Index  Altitudes 


0-149 

ICO 

4  50-249 

200 

25C-349 

3CC 

350-449 

4CC 

450-"’49 

3CC 

750-1 ,249 

1,000 

1 ,250-1 ,749 

1,500 

1 ,750-2,^99 

2,000 

2,500-3,499 

3,000 

3,500-4,499 

4,000 

4,500-7,499 

5,000 

7,500-12,499 

10,000 

,500  or  more 

15,000 

To  arrive  at  a  total  relative  worth,  V,  for  an  altitude,  the  highest 
individual  relative  worth  value  in  the  grouping  range  for  an  altitude  was 
recorded  on  the  Objective  2  (OTP  2b)  Tata  Consolidation  Form  (Appendix  I' 
summed  over  all  participants .  Thi3  total  relative  worth,  V,  was  then  used 
rank  the  altitv ies  during  analysis  . 

4 .4  OBJECTIVE  3 

4.4.1  JOP  3a 

4.4.1 .1  Tata  Collection 

Data  collection  for  this  OTP  was  conducted  using  the  categorical  judgm 
on  the  CAMPS  Display  Evaluation  form  contained  in  Appendix  E. 
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comments  . 

•i  .  ^  wBJSCT^T*  -• 

a.  =  .:  mo?  4  a 

J.  .5  •  *  *  *  Data  Oollection 

Data  collection  for  this  MDF  consisted  of  the  four  alternative  kr.ee 
oards  (Appendix  2)  ranked  in  order  of  preference  07  each  test  partioipa 

4.=  .*  .2  Data  Reduction 

The  Data  for  .'DP  4a  vas  reduced  by  recording,  for  each  participant, 
ranking  assigned  to  the  kneeccard  card  alternatives.  Dhe  Data  Heductio 
for  Objective  i  (MO?  4a)  vas  used  far  this  purpose  and  can  be  found  in 
Appendix  3 • 

i.=  .2  >DP  ib 

x. 5.2.1  Data  Collection 

Data  collection  for  /DP  4b  consisted  of  the  written  comments  of  the 
participants  concerning  the  format  ar.d  content  of  the  kneeboard  card 
alternatives  provided  to  then.  These  comments  were  made  on  the  kneeboa 
cards  themselves . 

4. 5> 2. 2  Data  Reduction 

Data  reduction  for  this  .'DP  vas  limited  to  the  consolidation  of  the 
participant  comments . 


•4 


ata 


DC?  consisted 
•rest  participants  as  indicated  on  the  Ease 
in  Appendix  E. 

A. 6. 2  Data  Reduction 


one  oategonoa.  juogr.ents  c:  t 
Planning  Cuestionnaire  contain 


The  data  reduction  for  vr 
participant  responses  cr. 
tally  of  the  circled  tarticioant  nun be 


c  ocr.sistec 


tn 9  recording  o:  tes* 
m a  — y  Sheer 


using  a 


l.~  c3ject:ve  5  ,  mc?  6: 

-.".1  Data  Dollecticr. 

Data  collection  for  this  objective  was  accomplished  using  the 
lomoarison  fora  containec  m  Accer.dix  E .  Particitants  ye  re  asked 


ICC 

points  between. 

each  pai 

on  t 

he  value  of 

the 

system 

you 

were  oompari 
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sv3  terns 

they 

would  each 

be 

assigned 

and 

A  of  little 

or 

no  value 

civile 

on  the  fora  bas 
For  examnle,  if 


3  was  considered  extremely  valuab 


:  oi-ows : 


*  DC 


<1.7.2  Data  Reduction 

The  data  for  this  ®?  was  recorded  on  the  Objective  5  (®?  o'  Data 
Reduction  Pom  contained  in  Appendix  3.  This  data  was  then  reduced  by 
using  the  Constant  Sum  Method  of  scaling.  Details  of  the  Constant  Sun  nethc 
are  contained  in  Appendix  ?. 

A. 3  OBJECTIVE  ^  (MO?  7) 

A. 3.1  Data  Collection 

Data  Collection  for  Objective  ~  consisted  of  the  categorical  judgment  of 
the  test  participants  as  indicated  on  the  Ceneral  Evaluation  fora  contained 
Appendix  E. 

A. 3. 2  Data  Reduction 

The  data  reduction  for  .MOP  7  consisted  of  recording  the  test  participant 
responses  on  the  Data  Summary  Sheet  for  Objective  7  ( MCP  7)  using  a  tally  of 
the  circled  tarticioant  number. 
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<1.9.2  Para  Reduction 

Data  reduction  for  this  objective  consisted  of  the  consolidation  of  test 
participant  comments  under  the  liked  and  disliked  categories  . 

1.10  PAPA  sur-a. 2ZIS 

Summaries  of  the  reduced  data  for  each  MP?  ere  contained  in  Appendix  3 • 
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'AT A  ANALYSTS 


5.1  OBJECTIVE  1 

The  JDPs  for  Objective  1,  as  specified  in  paragraph  2.;.‘,  were  analyzed 
using  the  same  method  of  analysis . 

5.1.1  Method  of  Analysis 


Since  each  test  participant  developed  flight  plans  using  CAMPS  and  mar.’ 
methods,  the  resulting  probabilities  of  survival,  ?(3),  could  be  organised 
into  matched-pairs  of  data  for  each  tesc  participant.  This  made  the  data 
ideally  suited  to  analysis  by  the  V ilcoxon  Matched-Pairs  3igr.ed-P.anks  Test 
(see  Siegel,  "Ncnparametric  Statistics,"  McGraw-Hill  *956). 


ual 


The  Vilcoxon  Signed-Panks  Test  focuses  on  the  difference  between  the  data 
elements  of  each  matched- pair .  It  considers  the  magnitude  of  the  difference 
as  well  as  the  direction,  thus  giving  more  weight  to  data  pairs  which  exhibit 
larger  differences  than  to  those  exhibiting  smaller  differences.  Tetails  of 
the  method  can  be  found  in  Appendix  H. 

5 .1.2  Assumptions 


The  Wilcoxon  Matched-Pairs  Signed-Panks  Test  is  a  ronparametric  technique 
and  thus  attributes  no  specific  distribution  to  the  data.  The  assumptions  for 
the  Signed-Hanks  Test  are  that  the  differences  being  analyzed  fora  a  random 
sample  from  a  distribution  which  is  continuous  and  symmetric . 

5.1.3  Hypotheses 

The  null  hypothesis  for  each  MOP  of  this  objective  is: 

CAMPS  planning  is  not  more 
effective  than  manual  planning. 


The  corresponding  alternative  hypothesis  is: 

CAMPS  planning  is  more  effective 
than  manual  planning. 


5 • 1 .4  Interpretation  of  Results 

After  the  Signed-Ranks  Test  was  applied  to  MOP  la  and  MOP  lb,  a  p- value 
for  each  null  hypothesis  resulted.  A  p-value,  or  significance  level,  is  a 
statement  of  the  probability  that,  if  the  null  hypothesis  is  rejected,  you  are 
in  fact  rejecting  the  true  case. 

Since  null  hypotheses  are  formulated  for  rejection  and  thus  the  implicit 
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acceptance  of  the  alternative  hypotheses,  it  i3  desired  tc  reject  the  null 
hypothesis . 

It  is  common  that  a  null  hypothesis  is  not  rejected  'unless  the  o-value  'or 
the  hypothesis  is  less  than  some  probability  prescribed  as  acceptable  ri3k  of 
rejecting  a  true  case.  This  level  of  risk  is  denoted  as  <  >  and  Typically  has 
values  of  .05  or  .1 . 

5.2  OBJECTIVE  2 

5.2.?  MOP  2a 


The  opinions  of  the  test  participants  regarding  the  minimum  number  of 
altitude  slices  required  were  analyzed  using  the  node  of  the  data  to  select 
the  number  of  slices  nost  preferred.  In  addition,  the  arithmetic  mean  and 
standard  deviation  for  the  data  were  computed  . 

5.2.2  ffiP  2b 


The  analysis  of  the  relative  worth  measure  for  this  MOP  consisted  of 
ranking  the  altitudes  by  relative  worth  from  highest  to  lowest. 

5.5  OBJECTIVE  5 

5.5.1  MOP  5a 


Analysis  of  the  categorical  judgments  for  this  MOP  consisted  of  computing 
the  percentage  of  test  participants  judging  a  display  characteristic  in  each 
category. 

5.5.2  MOP  5b 


The  comments  of  the  test  participants  were  summarized  and  are  included 
without  further  analysis  . 

5 .4  OBJECTIVE  4 

5.4.1  MOP  4a 

5. 4. 1.1  Mathod  of  Analysis 

Although  subjective  in  nature,  the  rankings  of  the  kneeboard  cards  by  the 
test  participants  for  MOP  4a,  were  analyzed  by  constructing  an  interval  scale 
based  on  the  ordinal  rankings  of  the  four  alternative  kneeboard  cards  'Olenn 
?.  Lindsay,  "On  Constructing  Interval  Scales  from  Ordinal  Judgments",  Maval 
Postgraduate  School,  1977,  unpublished). 

As  a  check  on  the  agreement  of  the  rankings  by  the  test  particioants ,  the 
Kendall  coefficient  of  concordance  was  calculated.  It  indicates  whether  the 
judges  applied  essentially  the  same  standard  in  ranking  the  various 
alternatives . 

Details  of  these  methods  are  contained  in  Appendix  H. 
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5.4..J.2  Assumptions 


The  assumptions  made  co  construct  an  interval  3cale  from  ordinal 
judgments  are: 

a.  A  judge  cannot  directly  express  his  feelings  about  the  scale  value  of 
an  alternative,  but  is  able  to  rank  the  alternatives  in  accordance  with  his 
feelings . 

b.  The  feelings  of  the  population  of  judges  are  a  normally  distributed 
random  variable . 

c.  The  variance  in  the  feelings  of  the  judges  is  the  same  for  ail 
alternatives . 

d.  The  correlation  coefficient  for  feelings  between  any  pair  of 
alternatives  i3  the  same . 

In  order  to  compute  Kendall's  coefficient  of  concordance,  the  only 
assumption  required  is  the  independence  between  rankings . 

5. 4..  1.5  Interpretation  of  Results 


The  interpretation  that  can  be  made  of  an  interval  scale  is  best  explained 
by  an  example.  If  four  instances  of  some  property  being  measured  are  denoted 
by  the  letters  3,  ?,  3,  and  X  (far  this  MDP  the  property  being  measured  is 
"user  compatibility'*  and  the  instances  are  the  four  kneeboard  card 
alternatives)  then  an  interval  measurement  of  the  degree  of  the  property 
possessed  by  the  instances  might  be  represented  as  follows: 

___  |  _  _  1  _________  _  __________  1  _  _  1  ____ 

?  3  H  3 

On  such  a  scale,  the  base  or  zero  point  as  well  as  the  'units  of  measure 
are  entirely  arbitrary.  It  could  be  concluded  that  3  and  H  possess  more  of 
the  property  than  3  and  ?,  that  there  appears  to  be  little  difference  between 
3  and  H  as  compared  with  the  difference  between  3  and  3,  etc.  If  the 
intervals  between  all  instances  were  approximately  equal,  then  very  little 
information  is  provided.  In  such  a  case,  since  the  'units  are  arbitrary,  the 
intervals  could  represent  large  but  equal  differences  or  small  but  equal 
differences . 

A  high  or  significant  Kendall  coefficient  of  concordance  may  be 
interpreted  as  meaning  that  the  test  participants  applied  essentially  the  3ame 
standard  in  ranking  the  kneeboard  card  alternatives.  Though  this  does  not 
assure  that  the  ranking  is  correct  (it  may  be  influenced  by  external  factors), 
it  does  give  confidence  in  the  ranking  under  the  conditions  of  the  test. 

5.4.2  JDP  4b 


The  comments  of  the  test  participants  regarding  the  kneeboard  card  were 
summarized  and  are  included  without  further  analysis. 
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5.5  OBJECTIVE  5 


The  method  of  analyzing  the  categorical  judgments  of  the  ease  of  planr.in 
of  the  OA.MFS  was  to  compute  the  percentage  of  test  participants  judging  a 
characteristic  in  each  category. 

5 .5  OBJECTIVE  6 

The  scaled  ranking  of  the  different  systems  resulting  from  data  reductic 
was  used  without  further  analysis . 

5.7  OBJECTIVE  7 

The  categorical  judgment  of  the  test  participants  as -to  their  overall 
opinion  of  the  CAiMPS/EPASS  were  analyzed  'ey  computing  the  percentage  of 
participants  judging  the  system  to  be  in  each  category. 

5.3  OBJECTIVE  3 

The  comments  of  the  participants  regarding  their  likes  and  dislikes  abou 
the  system  were  summarized  and  are  discussed  in  Section  5,  Results. 
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RESULTS 


6.1  OBJECTIVE  ' 

To  investigate  whether  a  plan  for  a  tactioal  air  strike 
mission  developed  using  CAMPS  results  in  a  higher 
probability  of  survival  than  one  developed  manually. 

In  order  to  meet  Objective  1 ,  the  difference  in  the  probabilities  of 
survival  for  a  mission  planned  with  CAMPS  and  a  manual  mission  was  used  to 
test  the  hypothesis  that  CAMPS  planning  is  not  more  effective  than  manual 
planning.  The  hypothesis  was  tested  for  two  parts  of  the  mission:  to  the 
target  and  for  the  entire  mission. 

6.1.1  Results 


Prom  the  analysis  for  Objective  1,  it  was  possible  to  reject  the 
hypothesis  that  CAMPS  planning  is  not  more  effective  than  manual  planning  with 
the  following  levels  of  significance: 

To  the  target  p  *  .032 
Total  Mission  p  »  .052 


6.1.2  Discussion 


The  hypotheses  for  Objective  1  were  designed  to  test  for  a  difference 
between  CAMPS  and  manual  systems  of  planning.  The  results  above  tell  us  that 
there  is  a  statistically  significant  difference  between  CAMPS  and  manual 
planning  methods.  That  is,  CAMPS  planning  provides  a  better  probability  of 
survival  than  manual  planning.  The  hypotheses  do  not  tell  us,  however,  how 
much  better . 

Ve  can  get  3ome  insight  into  the  magnitude  of  the  difference  by  looking  at 
the  means  of  the  probabilities  of  survival  for  each  method.  Table  6-1 
contains  the  mean  values  and  differences  of  the  ?(s)  for  the  conditions  of  the 
test.  If  we  consider  the  differences  as  improvement  and  compare  them  to  their 
respective  manual  ?(s),  we  get  a  percentage  of  improvement  for  both  measures 
of  performance: 


to  the  target,  26% 
and  for  the  mission 

The  probabilities  of  survival  and  their  differences  shown  in  Table  6-1  can 
be  viewed  from  a  different  perspective.  For  example,  if  ICO  aircraft  were  to 
fly  missions  planned  manually  and  1 0C  aircraft  were  to  fly  missions  planned 
with  the  CAMPS,  the  values  in  Table  6-1  tell  us  that  it  could  be  expected  that 
19  more  aircraft  would  make  it  to  the  target  and  16  more  aircraft  would  return 
from  the  mission  when  using  the  CAMPS  than  when  planning  is  done  manually. 
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Table  6-* 

.  Mean  Prcbabili 

ties  of 

Survival 

SAMPS 

Manual 

2t fference 

To  Target 

.922 

.”34 

.138 

Mission 

.324 

.561 

.  1  63 

6 .2  OBJECTIVE  2 

To  assess  test  participant  attitudes  regarding  the  number 
of  required  altitude  slices  and  the  specific  altitudes 
preferred . 

For  this  objective,  test  participant  opinions  were  used  to  assess  the 
number  of  altitudes  required  and  the  relative  worth  of  various  altitudes. 

6.2.1  Results 


Table  6-2  contains 
various  altitude  slice 


the  percentage  of  test  participants  selecting  the 
availabilities.  Statistics  for  this  data  are: 


Mode  ■  5 

Arithmetic  Mean  *  5 .39 
Standard  Deviation  *  1  .66 


Table  6-3  contains  the  index  altitudes  ranked  according  to  their  relative 
vorth  as  computed  during  data  reduction. 


Table  6-2.  Minimum  dumber  of  Altitude  Slices  Required 

'dumber  of  Percent 

Slices  Selecting 


3 

4. 

5 


3 

9 

10 


0 

16 

37 

21 

10 

5 

5 

5 
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Table  5-3*  Hanking  of  A1 tirades  by  Relative 


Hanking 

Index  Altitude 

Relative 

1 

500 

51  .354 

2 

1  ,  CCO 

29.0~5 

100 

25-565 

4 

200 

21  .356 

5 

500 

10.412 

6 

5,000 

9.454 

1 

10,000 

9.139 

3 

1,500 

5.989 

9 

15,000 

5 .5"2 

10 

5,000 

4. '”’9 

1 1 

2,000 

4.129 

12 

400 

4.04 

15 

4,000 

.353 

Worth 

Worth 


6.2.2  Discussion 

The  results  obtained  for  the  minimum  number  of  altitude  slices  tell  us 
that  at  least  4  are  required  and  probabl7  no  more  than  7  would  be  necessary . 

As  to  what  these  4  to  7  altitudes  might  be,  we  look  to  a  plot  of  the 
relative  worths  of  the  altitudes  by  rank  (Figure  5-1 )  .  This  plot  reveals  two 
clusters  of  the  data.  This  would  indicate  that  the  4  highest  ranking 
altitudes 


500 

1,000 

100 

200 

might  be  selected  with  some  confidence  over  those  of  the  lower  group,  the 
ranking  of  altitudes  within  a  cluster,  especially  the  lower  one,  should  be 
considered  more  variable  and  subject  to  interpretation . 
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Figure  6-1.  Floe  of  Relative  Worth  of  Index  Altitudes  by  Rank 


o 


To  assess  test  participant  attitudes  regarding  the  physical 
appearance  and  information  content  of  the  CAMPS  display. 

6 .3 .t  Results 

The  results  for  this  objective  are  summarized  in  Table  5-1  and  in 
Appendix  3  [.Test  Participant  Comments  -  Objective  3  ^ 'CP  3b).. 


Table  5-1 . 

CAMPS 

'4 

\  . J 

Display  Categorical 
responding) 

Judgments 

Property  Poor 

Only  Pa 

ir  Good 

Excellent 

Color 

3 

50 

12 

Information 

Content 

1 

38 

58 

Line  Textures 

13 

58 

29 

Screen  Size  1 

21 

57 

3 

Map  Scales 

1 

58 

33 

Overall 

50 

10 

5.3«2  Discussion 

Key  aspects  of  the  CAMPS  display  that  several  test  participants  wanted 
changed  were: 

1)  screen  size  (wanted  it  larger), 

2)  ability  to  change  altitude  display  during  planning  of  a  nission, 

3)  computer  response  too  slow. 

Other  comments  are  provided  in  Appendix  0  without  further  discussion. 

6.1  OBJECTIVE  1 

To  assess  the  user  compatibility  of  the  current  CAMPS 
kneeboard  card . 

6.1.1  Results 

The  analysis  of  the  rankings  of  the  kneeboard  card  alternatives  resulted 
in  the  following  scale  values: 
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Kendall  3  Coefficient  of  Concordance  resulted  in  the  rejection  at  the  . 
level  of  the  hypothesis  that  the  judges  used  different  criterion  to  rank  th 
kneeboard  card  alternatives  . 

Comments  on  the  various  characteristics  of  the  cards  are_ contained  in 
Appendix  G  jDest  Participant  Comments  -  Objective  A  'CCP  Ab).. 

5.1.2  Discussion 

Arranging  the  alternatives  on  a  scale  in  accordance  with  their  resultan 
scale  value,  it  can  be  seen  that  there  seems  to  be  little  difference  betvee 
alternatives  C  and  D  and  that  alternative  A,  the  current  kneeboard  card,  is 
least  oreferred  of  all . 


5.5  OBJECTIVE  5 


?o  assess  the  ease  of  planning  with  CA.MPS. 


In  order  to  accomplish  this  objective,  the  test  participants  were  asked 
rate  the  CAiMPS  in  four  areas  of  planning  and  then  rate  it  overall.  The  fiv 
statements  are  shown  in  Figure  6-2  • 


1 .  Hate  how  easy  it  was  for  you  to  get  information  you  needed  from 
the  system. 

2.  Rate  how  easy  it  was  for  you  to  use  the  information  provided  by 
the  system  to  plan  your  mission. 

3.  Rate  how  easy  to  understaand  and  execute,  were  the  procedures  for 
using  the  system. 

4.  Rate  how  well  the  system  provided  the  level  of  detail  of 
information  you  required . 

5.  Considering  your  responses  to  the  above,  rate  the  overall  ease  of 
planning  a  tacitcal  air  strike  flight  plan  with  the  CA.MPS . 
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'  V  ‘ 


.ac_e  o-: 


4. 

Overall 

5 .5  C3J3CTI73  5 


iase  a:  r.annin 
Cnly  Fair 

3 


.g  Taregcri 

cal  Judgment  F.esul 

Good 

Excellent 

38 

53 

42 

42 

34 

38 

o2 

value  of 

adding  Fxcenme: 

i  Support 

System  ,3?AS3. 

canabilities  to  the  CAMPS . 


This  objective  was  set  by  having  the 
3ystems  (i.e.,  CAMPS,  CAMPS/3PASS, '  3PASS) 
value  to  the  mission  planning  process  . 


test  participants  compare 
to  each  other  in  regard 


the  various 
to  their 


6.6.1  Results 

The  results  for  this  objective  vere  the  scale  values  shown  below  for  each 
of  the  systems. 

CAMPS/ 3?  ASS  2.15 

CAMPS  1 .05 

SPASS  .44 

6.6.2  Discussion 

What  the  results  above  tell  us  is  that,  in  the  opinion  of  the  test 
participants,  CAMPS  has  twice  the  value  of  SPASS  in  mission  planning  but  that 
CAMPS  with  3PASS  would  have  twice  the  value  of  CAMPS  alone .  Thus  the  marginal 
value  of  adding  SPASS  to  CAMPS  is  as  great  as  the  value  of  CAMPS  alone . 

6.7  OBJECTIVE  7 


To  assess  the  user’s  overall  opinion  of  the  CAMFS/3PASS 
system  as  a  mission  planning  tool. 

In  order  to  achieve  this  objective,  test  participants  were  asked  to 
respond  to  the  following: 

’What  was  your  overall  impression  of  a  CAMPS/ 3? ASS  type  of  system 
as  a  tool  for  :.ission  planning? 

Poor  (unimpressed)  _ 

Pair  _ 

Good  _ 

3mcellent  (very  impressed)  _ 
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’.’  CCNCLUSICNS 
’•1.’  Effectiveness 


Eased  on  the  results  of  this  test  and  the  previous  test  that  used  CAMPS, 
the  conclusion  can  be  drawn  that  the  CAMPS  is  a  valuable  mission  planning  aui. 
It  should  be  pointed  out  however,  that  the  results  of  both  of  these  tests 
depended  upon  the  level  of  threat  and  the  perfect  intelligence  assumptions. 

The  fact  that  nan y  participants  achieved  probabilities  0:'  survival  of  ‘  with 
both  methods  of  planning  also  reduced  our  ability  to  discrimnate  between  the 
systems  with  accuracy.  These  limitations  do  not  lessen  confidence  in  the 
results  but  reduce  the  ability  to  generalise  the  results  to  other  threat  and 
intelligence  reliability  levels. 

The  value  of  an  automated  system  would  vary  as  the  threat  varies.  That 
is,  when  there  is  little  or  no  threat,  there  would  be  little  or  no  difference 
in  missions  planned  with  either  system.  As  the  intensity  of  the  threat 
increases,  it  would  be  expected  that  the  value  of  the  CAMPS  would  increase 
until  a  point  is  reached  where  the  threat  is  30  dense  that  missions  claimed 
with  perfect  knowledge,  whatever  its  source,  would  be  flown  at  extremely  high 
risk. 

The  sensitivity  of  the  effectiveness  of  the  CAMPS  to  changes  in  the  level 
of  threat  and  intelligence  reliability  is  unknown.  This  does  not  iiminisr.  the 
potential  value  of  the  CAMPS  to  the  mission  planner,  but  only  our  ability  to 
predict  its  effectiveness. 

".1.2  Utility 

On  the  whole,  the  vast  majority  of  the  test  participants  felt  the  CAMPS 
would  be  invaluable  to  them.  They  felt  that  it  provided  them  with  tools  not 
currently  available  to  them  and  that  these  tools  were  easy  to  use  and  apply. 

Their  indications  of  deficiencies  and  suggestions  for  improvements  are 
detailed  in  Section  5  (RESULTS)  and  in  Appendix  0.  Erom  these,  it  can  be 
concluded  that  the  CAMPS  is  on  the  right  track.  It  provides  a  needed 
capability  to  the  mission  planner  and  is  easy  to  use. 

7.2  RECOMMENDATIONS 


As  with  most  tests  or  experiments,  more  questions  result  than  are 
answered.  The  CAMPS  Test  ’-62  is  no  exception. 

The  key  questions  to  be  answered  in  the  future  are: 

1 .  Vhat  is  the  sensitivity  of  CAMPS  performance  to  different  levels  of  enemy 
threat? 

2.  How  will  real  intelligence  (i.e.,  reliabilities  less  than  -QC%)  affect  the 
performance  and  utility  of  the  CAMPS. 
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APPEND II  A 


PLIGHT  3 CUTS  EVALUATION  PROCEDURES 


A .  1  INTRODUCTION 

In  order  to  compare  the  flight  routes  prepared  by  participants  using  CANPS 
and  manual  planning  methods,  the  routes  were  entered  into  a  00 NARCO,  Inc., 
system  called  SPASS  (Experimental  Penetration  .Analysis  Support  System) .  The 
SPASS  simulates  the  flying  of  a  mission  through  a  specified  threat  over 
specified  terrain.  .Among  its  outputs  is  a  probability  that  a  particular 
engagement  of  a  surface-to-air  missile  (SAN)  will  inflict  abort-level  damage 
to  the  aircraft  flying  the  mission.  The  next  paragraph  outlines  the 
procedures  used  to  evaluate  the  flight  routes  using  the  EPASS. 

A. 2  EVALUATION  PROCEDURE 

A.  Log-on  to  CONARCO,  Inc.  EPASS  Program 

3.  Designate  for  scenario 

1 .  Terrain 

2.  Air  Defenses 
3 •  Aircraft 

C .  For  each  flight  route 

1 .  Set  up  a  Data  Collection  and  Reduction  Form 

a .  Pilling  in  Participant  Number 

b.  Circle  method  of  planning. 

2.  Enter  all  tumpoints/' checkpoints  of  route  in  sequence  by  entering: 

a.  Coordinates 

b.  Altitude 

c .  Speed 

3.  Evaluate  route  by  using  SPASS  simulation. 

4.  Display  engagement  list. 

5.  Record  following  data  on  Collection  and  Reduction  Porn. 

a.  Sites  firing  (column  2) 

b.  Time  of  engagement  (column  3) 

c.  Probability  of  damage,  ?(d)  for  each  firing  (column  5)* 

d.  Time  on  target  (TOT) 

*  NOTE:  Column  4  of  the  Data  Collection  and  Reduction  Fora  is  not  used. 
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APPENDIX  3 
SCENARIO 


3.1  INTRODUCTION 


The  scenario  for  the  CAMPS  Test  1  -32  was  very  Halted  but  specifically 
designed  to  provide  an  environment  and  situation  to  test  the  particular  0 AMPS 
capabilities  in  question. 

Participants  were  not  asked  to  develop  complete  flight  plans  in  that  they 
had  no  fuel  or  time  calculations  to  consider.  They  only  had  to  develop  the 
route  itself.  In  order  to  structure  the  test  environment,  the  planning 
constraints  described  below  were  developed. 

3.2  DESCRIPTION 

The  situation  used  for  this  test  was  based  on  a  scenario  used  at  the  VTI 
course  held  at  MAWTS-1 ,  Yuma,  Arizona. 

3.2.1  Planning  Region 

The  region  made  available  to  the  participants  for  planning  is  shown  below: 


3.2.2  Missions 

There  were  two  missions  planned  by  each  participant .  Each  mission 
consisted  of  a  designated  start  point  and  a  target.  The  participant  was 
"placed"  in  the  air  at  the  start  point  and  required  to  develop  a  route  to  .he 
target  and  back  to  the  start  point.  The  two  missions  are  described  below: 
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.MISSION  *' 


START  POINT: 

"5  "5  ^ 

>  ^  wVrf  .1 

1 1 4°45 '  V 

•c,:cc  ?T  .'EL 

TARGET: 

32°31 '30" 

N 

1 12°55’45" 

V 

MISSION  #2 

START  POINT: 

32°1 5  ’  N 

1 1 4°45 '  W 

10,000  ?T  MSL 

TARGET: 

32a43’  N 
112°51 *  V 

3.2.3  Profile  Constraints 

In  order  to  simulate  the  effects  of  fuel  and  range  limitations  that  would 
normally  apply  to  mission  planning,  participants  were  required  to  adhere  to 
the  certain  mission  profile  constraints. 


Altitudes  and  the  distance  that  could  be  flown  at  an  altitude  were  limits 
so  that  the  entire  mission  could  not  be  flown  at  any  one  altitude  because  of 
fuel  limitations.  The  limitations  on  altitude,  speed,  and  distance  are  shown 
below. 


Altitude  ( 

ft) 

Mhximum  Cumulative  Distance 
at  or  below  Altitude  (MM) 

Ground 

Sneed  (Kls) 

10,000 

MSL 

300 

420 

1  ,000 

AGL 

150 

4S0 

500 

AGL 

50 

4  30 

100 

AGL 

30 

540 

For  example, 

if 

30  MM  are  flown  at  100  ft  AGL,  then 

only  30  MM  could  be  f' 

at  500  ft  AGL. 


Total  Round  Trip  Distance  for  a  route  had  to  be  3CC  N M  or  less. 

3.2.4  Threat 

Enemy  SAM  sites  used  for  the  test  were  also  based  on  the  WTI  Scenario. 
The  type  and  location  of  the  sites  are  listed  below: 

SURFACE-TO-AIR  .MISSILE  THREAT 

TYPE  LOCATION 

SA3  32  39  00  N 

112  36  00  V 

SA4  32  24  13  M 

112  4?  4C  V 
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55 

35 
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¥ 
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00 
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32 
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45 

48 
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16 

1 1 

N 
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44 

1 1 

¥ 
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14 

32 

7 

112 

56 

41 

¥ 
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00 
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X 
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50 

CO 

7 

113 

H 

X 

7 
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32 
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20 

7 
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06 

X 

V 

3A8 
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49 

06 
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54 

41 

¥ 
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32 
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7 
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37 

36 
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112 

52 
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¥ 
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40 

30 
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37 

X 

¥ 
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TEST  PERIOD  OUTLINE 


C.1  OUTLINE 

I.  Welcome  Aboard  i  Introduction 


A.  INTRODUCTIONS 

3.  PURPOSES  —  The  purposes  of  this  test  are  two- fold : 

(l  )  To  see  if  an  automated  system  3uch  as  this  will  aid  in  the  flig 
planning  of  tactical  missions . 

(2)  To  expose  as  many  aviators  as  possible  to  the  system  and  make 
them  aware  of  the  potential  capabilities  • 

Right  now,  no  aviators  are  involved  in  its  acquisition .  We  want  to 
find  out  if  the  aviation  community  has  a  need  for  a  system  like  this  and  if 
get  them  involved  in  its  acquisition  by  generating  a  ground  swell  of  opinion 
from  the  FMF  and  thus  encouraging  the  designation  of  an  aviation  snonsor  at 
HQMC . 


C.  BACKGROUND 

'What  is  CAMPS?  (Show  display  slices) 

'Who  developed  it?  Sterling  Engineering  Division  COMARCC,  Inc. 

When  we  get  USMC  system  i  how  many. 

This  system  is  on  loan  from  CO  MARCO  to  the  U.S.  Navy  who  in  turn  has  loaned 
to  the  USMC  for  use  at  WTI  1-32  and  for  this  test. 

D.  WHAT  YOU'LL  3E  DOING 
MISSION  BRIEFING 
FLIGHT  ROUTS  PLANNING 

CAMPS 

MANUAL 

DEMONSTRATION  (CAMPS/EPASS) 

DEBRIEF 

II .  MISSION  3RIEFING 

Situation  and  Mission 

General  Orientation 
Overall  Mission 


Enemy  Threat  ,3-2' 

Mission  Parameters  and  Aircraft  Pyre 
Target  Assignments 

III.  PLANNING 
Period  1 . 

Period  2. 

IV.  C AMPS/ 2P ASS  DEMONSTRATION 

V.  DEBRIEF 

Questionnaires 

Comments 

C  .2  SCHEDULE 


Welcome  &  Mission  3rief 

Planning  Period  1 

Planning  Period  2 

Demonstration 

Debrief 

End 
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1000 

1030 


II 
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1100 

1200 

1300 

1330 


III 

1330 
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1500 

1600 
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1430 


1730 
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DATA  COLLECTION  AND  REDUCTION  TORES 


OBJECTIVE  1 

CAMPS  Xneeboard  Card  . 

Manual  Flight  Route  Tom  . 

Route  evaluation  Data  collection  &  Reduction  Tom 
Objective  1  ( MOP  la)  Data  Summary  Sheet  ....... 

OBJECTIVE  2 

Altitude  Slice  Questionnaire  . 

Objective  2  (MOP  2a)  Data  Summary  Sheet  . 

Objective  2  (MOP  2b)  Data  Reduction  Tom  . 

Objective  2  (MOP  2b)  Data  Consolidation  Tom 
OBJECTIVE  3 

CAMPS  Display  Evaluation  . 

Objective  3  (MOP  3a)  Data  Summary  Sheet  . 

OBJECTIVE  4 

Kneeboard  Card3  (A  through  D)  . 

Objective  4  (.MOP  4a)  Data  Reduction  Sheet  . 

OBJECTIVE  5 

Ease  of  Planning  Questionnaire  . 

Objective  5  (MOP  5)  Data  Summary  Sheet  . 
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OBJECTIVE  5 


OAMPS/EPASS  Comparison  . 

Objective  5  ( >50P  5)  Tata  Reduction  ?:n 

OBJECTIVE  7 

General  Evaluation  . 

Objective  7  (MOP  7)  Data  Summary  Sheet 

OBJECTIVE  3 

General  Evaluation  . 
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] .  Race  how  easy  it  was  for  you  to  gat  the  information  you  needed  from  the 
system. 

Terrible  Poor  Inly  fair  Pood  Excellent 

-•  Rata  .tow  easy  it  was  tor  you  to  use  the  information  provided  by  rha  svstam 
to  plan  your  mission. 

Terrible  Poor  Only  fair  Pood  Excellent 


3.  Rata  how  easy  to  understand  and  execute ,  were  the  procedures  for  using 
system. 

Terrible  Poor  Only  fair  Good  Excellent 


4.  Rata  how  well  the  system  provided  the  level  of  detail  of  information  you 
required. 

Terrible  Poor  Only  fair  Good  Excellent 


Considering  your  responses  to  the  above,  rate  the  overall  ease  or  planning 
a  tactical  air  strike  flight  plan  with  the  CbdfPS . 

Terrible  Poor  Only  fair  Good  Excellent 

Comments : 
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PASTS 


CIPANT  MC. 


camps /s? ass  comparison' 


CAMPS 


SPAS  5 


CAMPS /SPAS 3 


CAMPS /SPAS 3 


CAMPS 


SPASS 


COMMENTS : 


E-20 


e-:i 


participant  no. 


fTTNTRAl  gTALCATICN 


What  was  your 
a  tool  for  si s 


over  an  iiroression 
sion  ol  amine? 


3-  a  CAM0S/S?ASS  tyoe  o* 


system  as 


Pcor(unimoressed) 


Fair 


'  food _ _ 

excel! ent( very  itnoressed) 


What 


did  you  like  .nest  aaout  the  CAMPS/;?AS$  system? 


|  What  did  you  dislike  about 

r 


the  CAMPS/SPAS3 


system? 


E-22 


i 


E-23 


?he  constant  sum  metpcd  rcR  scxlz'm 

Reference:  Clenn  ?.  Lindsay,  "Scaling  with  the  Constant  Sun  Method" ,  '7aval 

Postgraduate  School,  I960. 

The  Constant  Sum  method  was  used  for  Objective  6  to  scale  *he  cronertv 
"value  to  the  mission  planning  process"  and  locate  the  instances:  CAMPS , 
CAMPS/EPASS,  and  EPASS ,  on  the  resulting  scale.  This  method  reouires  that  the 
participants  agree  upon  an  origin  for  this  scale  and  this  will  be  assumed  . 

The  method  will  result  in  a  ratio  scale,  “"he  following  orocedures  adanoed 
from  the  reference  were  used  . 

STEP  1  Each  test  participant  ’  s  point  assignments  recorded  on  the  '’AM?5 /S,?A?S 
comparison  form  were  transferred  to  the  Objective  o  ■  M"1?  o'1  Tata 
Reduction  Porm  (Appendix  3)  as  follows.  The  point  assignments 
corresponding  to  the  circled  numbers  below  were  recorded  in  the 
corresponding  column  for  the  test  carticinant  on  the  Tata  ’eduction 
Porm . 


CAMPS  CD 
EPASS  Cp 
CAMPS/EPASS  0 


CAMPS/EPASS 

CAMPS  0 
EPASS  © 


STEP  2  Compute  the  mean  of  each  column  of  the  Data  Reduction  Porm. 

STEP  3  Record  the  means  (labeled  @  -  (f)  1  from  the  Data  Reduction  Porm  into 

the  corresponding  cells  of  the  following  array.  The  means  become  the 
elements  a^  of  the  matrix  A. 


camps  camps/ 

EPASS 


SPASS 


CAMPS 


-  CAMPS/ 

‘  EPASS 


EPASS 


A  MATRIX 


7-1 


STEP  4  Compute  a  matrix  V,  with  elements 


Vij 


a .  . 

Ji 


STEP  5  Compute  scale  values  3.,  where 


are  the  geometric  column  means . 


APPENDIX  3 
DATA  SUMMARIES 


OBJECTIVE  1 

3DP 

MOP 

OBJECTIVE  2 

MOP 

MOP 


OBJECTIVE  3 

MOP 

MOP 

OBJECTIVE  4 

MOP 

MOP 

OBJECTIVE  5 
MOP 

OBJECTIVE  5 
MOP 

OBJECTIVE  7 
MOP 

OBJECTIVE  3 
MOP 


la  —  To  the  Target 
1  b  —  For  the  Mission 


2a  —  Number  of  Altitude  Slices 
2b  —  Relative  Worths  of  Altitudes 

Listed  Altitudes  . 

Index  Altitudes  . 


3a  —  Display  Characteristics  . 

3b  —  Consolidated  Comments  on  the  CAMPS  Display 


4a  —  Kneeboard  Card  Rankings  . 

4b  —  Consolidated  Comments  on  the  Kneeboard  Cards 


5  —  Ease  of  Use 


6  —  CAMPS/SPA SS  Comparison 


7  —  Overall  Impression  of  CAMPS/3PASS 


3  --  Consolidated  Commerts  on  CAMPS/EPASS  System 
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Note:  If  more  than  one  participant  made  a  comment,  the  number  is  indicated 

in  parentheses  following  the  comment. 

1.  The  display  is  an  effective  aid. 

2.  Would  like  to  have  the  cursor  stay  visible  and  be  able  to  move  it  in 
real- time . 

3.  The  display  should  be  larger.  ■ o' 

i .  Display  fuel,  distance,  etc.,  for  a  candidate  point. 

o .  Be  able  to  change  altitude  slice  display  while  planning  initial  route. 


(5) 

5.  Have  display  change  automatically  when  altitude  is  changed. 
n  .  Display  leg  altitudes  on  route. 

3.  Labeling  cluttered  screening,  may  be  related  to  3ize . 

9.  Need  a  color  legend. 

10.  Use  letters  for  checkpoints  vice  numbers. 

11.  Need  terrain/map  features  to  aid  in  orientation,  (i) 

Note:  The  system  has  the  ability  to  add  many  features.  This  capabil'  — 
used  only  to  a  very  limited  extent  for  the  test .  Increasing  displays 
features  also  increases  clutter. 

12.  Course  line  too  broad  or  blocks  too  big  to  distinguish  their 
intersection. 


13-  Would  like  to  have  point  location  of  threats  without  coverage. 

1 4 .  Be  able  to  shift  threat  scales  more  easily. 

15-  Be  able  to  print  route  on  a  map . 

1 6.  Make  encyclopedic  information  on  aircraft  characteristics,  weapons, 
maneuvers  to  degrade/def eat  weapons,  etc.,  available  in  the  system. 


O.  \ 


OBJECTIVE  ~  '"C?  -b 

CONSOLIDATED  COMMENTS  ON  THE  OEE3CAHD  CASES 

Note:  If  more  than  one  carticipant  made  a  comment,  one  number  is  indicated 

in  parentheses  following  the  comment. 

1.  Spread  the  information  out. 

2.  Put  only  6-7  points  per  card.  (2) 

3.  Include  True  Course  and  Magnetic  Course  for  the  A-6 .  (3) 

I .  Meed  larger  print. 

5.  Eelete  NAVAID  information  since  not  available  in  combat.  '.?) 

6.  Use  JBL  only. 

7.  Show  speed  changes  only. 

3.  Space  horizontally  more. 

9-  Like  actual  fuel  notation  column. 

10.  Don't  need  ETA/ATA  or  actual  fuel  column,  (4) 

II.  Use  JBL  for  altitudes  above  1CC0  ft.  and  ACL  for  altitudes  below  ‘ ECC  f 

12.  Use  only  one  altitude  entry,  ACL  or  JBL,  not  both. 

13.  Like  the  bingo  fuel  entry.  (3) 

14.  Want  checkpoint  plain  language  identifier. 

1 5 .  Reverse  ETS  and  cumulative  times . 

16.  A-4  pilots  don't  need  kneeboard  cards.  Information  must  be  transferred 
to  map-.  Therefore,  format  ,isn' t  critical..  (2) 

17.  Put  card  information  on  heads-up  display  (HUD) 

18.  Like  highlighting  of  various  items  of  information.  (3) 

19.  Don't  need  magnetic  variation.  (5) 

20.  Need  magnetic  variation. 

21.  Want  true  air  speed  instead  of  ground  speed.  (2) 
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CONSOLIDATE)  COMMENTS  ON  CAMPS/SPASS  SYSTEM 

Note:  If  more  than  one  participant  aade  a  consent,  the  number  is  indicated 
in  parentheses  following  the  comment . 

LIKES 

1.  Saving  of  manual  calculations.  (5) 

2.  Interpretation  of  terrain  effects. 

3-  Mission  information  printout.  (3) 

4.  Threat  display  at  the  different  altitudes.  (14) 

5-  Target  area  display. 

6.  Ability  to  test  alternative  SCM  options  on  SPASS .  (2) 

7.  Flexibility  of  the  system  while  planning.  (2) 

9.  Easy  to  operate.  (2) 

9.  SPASS  probability  charts. 

10.  EPASS  threat  breakdown  by  site. 

DISLIKES 

1.  Sensitivity  to  intelligence  spoilage.  (6) 

2.  Unknown  terrain  accuracy.  (Unverified) 

3.  Slow  response  of  system.  (4) 

4.  System  too  bulky. 

5.  Doesn't  display  major  terrain.  [See  note  for  Comment  11,  Objective  3 
(MOP  3b)] 

6 .  Concern  about  ruggedness  . 

7.  Screen  too  small.  [See  Comment  3,  Objective  3  (MOP  3b)] 

8.  Not  available  at  squadrons.  (3) 

9.  Would  not  be  responsive  enough  for  CAS  on-call  missions  since  the  threa 

changes  so  fast . 

10.  Didn’t  like  touch  panel  menu. 

11.  Lack  of  correlation  between  display  and  map. 


12.  Dial-iked  the  limitation  of  only  one  candidate  porno  at  a  tine 

13*  Wanted  to  be  able  to  change  altitude  3lioe  display  while  plan 
[See  Comments  5  and  5  of  Objective  3  (-&?  3’o)j  (2) 

14.  Felt  there  was  some  confusion  in  operating  system. 
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appendix 


Tab  1 

Tab  2 

Tab  3 


DETAILS  D?  ANALYSIS 


—  Wilcoxon  Matched-Pairs  Signed-P.anks  Test 

—  Interval  Scales  from  Ordinal  Judgments 

—  Kendall's  Coefficient  of  Concordance 
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VILCOXOM  yATCHXP-PAIP.S  SIC-NZD-RANX  TEST 
Summary  of  the  Procedure 

1.  For  each  teat  participant  determine  the  signed  difference,  between 

the  probability  of  survival  for  CAMPS,  ?^(s),  and  the  probability  of 

survival  for  manual  missions,  ?  's)  . 

a 

2.  Panic  the  differences  without  regard  to  the  sign.  Smallest  difference  is 
ranked  first,  largest  is  ranked  last. 

3  •  Replace  the  signs  . 

4.  Compute  2  ■  sum  of  the  positive  ranks. 

5.  Determine  N  *  number  of  differences,  D.  ,  not  equal  to  oero . 

6.  Use  T  and  N  for  table  look-up  of  significance  level. 

Table  Reference:  DIXON-. MASSEY,  Introduction  to  Statistical  Analysis, 
McGraw-Hill .  ?.  543  Table  A- 1 9  - 
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TA2  2-o  a??e:ioix  h 

INTERVAL  3CA1I2  FRC.'T  CROI^Al  JUTS  .VENTS 

Test  participant  ordinal  responses  are  tallied  into  an  f\  4  array  whe: 
each  element,  f- 4,  is  the  number  of  tarticipant3  who  ranked  alternat: 
above  alternative  i,  i.e., 

j  »  A  3  2  3 

A  —  f  „ 

A3  *AC  * A3 


f  * 

CA  "C3 


‘3A  3B  *3C 


A  p.  4  array  is  construct 


ed  where 


,  *  f 
•ij  Ji 

3iagonal  elements  of  the  p. .  array  are  set  to  0.5- 

3-  A  Z  .  array  is  then  computed  where  each  Z.  .  element  is  the  standard  normal 
percentile  corresnonding  to  the  p„.  . .  For  .02  >  p.  .  >  .98,  the  Z-  .  cell  i3 

X  J  —  O  O 

left  empty. 

4- .  If  the  Z.  .  array  contains  no  empty  ceils,  then  the  column  averages  can  be 

*  v 

used  as  scale  values,  S4,  for  the  alternatives. 


S.  •  Z  2.4  ,  j  -  A, 3,0,3 
J  i-A 


5.  If  the  3.  .  array  has  empty  cells,  then  a  least  squares  method  must  be  used 
•  J 

for  columns  with  empty  cells.  For  complete  columns,  the  column  average  may  be 
used  as  the  scale  value.  For  those  columns  with  empty  ceils,  3  set  of  linear 
equations  of  the  form, 
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MICROCOPY  RESOLUTION  TEST  CHART 

NATIONAL  BURtAu  0»  STANDARD*  1%:  A 


TA3  y  to 

KZNDALL '  S  CCSSTICIZaT  0?  CONCORDANCE 
Summary  of  Procedure 

1  .  The  columns  of  the  Objective  4  (MOP  4a)  lata  Summary  Sheet  were  3ummed  to 
yield  the  values  5..  for  j  *  A,3,C,.C. 

U 

2.  Compute  R,  where 

R  -  - - 

3.  Calculate  the  sum  of  squared  deviations. 

s  -  £  (3.  -  H)2 

J 

4 .  Calculate 

x  -  k2(y3-;p  -  2sao 

12 

where  k  *  number  of  judges  ■  24 

N  -  number  of  alternatives  ranked  *  4 

5.  Compute  the  Kendall  Coefficient,  V 

,.  a  s 

*m~TT  2553 

5.  For  small  samples,  JI  _<  7,  the  Kendall  coefficient  is  tested  for 
significance  as  follows: 

If  the  observed  s  is  equal  to  or  greater  than  that  shown  in  Cable  3  of 
Siegel,  then  the  hypothesis  of  independent  rankings  may  be  rejected  at  the 
particular  level  of  significance . 
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